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ABSTRACT 

This paper uses results of postal surveys carried out 
in the United Kingdom further education (FE) and university sectors 
to highlight themes and elaborate on suggestions for advancing 
performance measurement. Academic libraries face insecurity arising 
from the political and educational changes taking place in academic 
institutions, competition between institutions and services, and the 
demands for accountability at a variety of levels within 
institutions. As a result, it is essential that there are mechanisms 
in place to monitor and evaluate library services and to maintain the 
high quality demanded by the academic community. When surveyed, 
librarians' views on performance assessment followed three main 
themes: the use of qualitative data in addition to quantitative data; 
user expectations and how to satisfy the information needs of an 
increasingly diverse academic community; and the climate of review 
and assessment, where periodical reviews can act as effective 
motivators in highlighting and encouraging reflection on how 
effectively the library supports teaching and research. The following 
practical suggestions for improvements in performance measurement are 
discussed: (1) greater involvement of a variety of constituents; (2) 

more systematic and structured approach; (3) greater proactivity; (4) 
wider application of information technology; (5) greater emphasis on 
the user perspective; and (6) more time. (Contains 10 references.) 
(Author/ SWC) 
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T he aim of this paper is to use the results of 
postal surveys carried out in the UK further 
education (FE) and university sectors to highlight 
themes and elaborate on some of the suggestions for 
taking performance assessment forward. 

Introduction 

INSECURITY 

The whole issue of assessing the quality of service 
we provide in academic libraries needs to be 
explored against a background of change - both in 
further and higher education - of frightening propor- 
tions. These are some of the more important ones: 

- Institutions have merged (and continue to do so) 

- Incorporation of polytechnics and colleges 
- Pol 5 dechnics are now universities 

- Student numbers have increased enormously 
(70% over 6 years) 

- Greater diversity of student population 
- Courses have been modularised 
- Interdisciplinary studies have grown considerably 
- Postgraduate education has expanded 

- Teaching and learning methods are changing 
(more independence and choice, less direction 
and contact time) 

- Funding mechanisms have altered 

- Institutional governance has become more 
managerial 

- A heightened culture of assessment is evident 

These changes - and I make no apology for listing 
them here - have produced a climate of insecurity. 
As Heery (1995) rightly points out: 

‘Many librarians express a nostalgia for the previ- 
ous certainties that seem now to be under threat.’ 
Or at least they are looking for a period of consoli- 
dation. 

COMPETITION 

These above changes plus the considerable techno- 
logical developments have produced a culture of 
competition between institutions and the services 



provided. Technology is stUl sexy and has the power 
to attract students. It is a marketable product and 
there is a rising expectation that any institution 
worth its salt wUl demonstrate (via the library or 
more widely) its desire to be at the leading edge. 

These developments include: 

- SuperJANET 

- Metropolitan Area Networks (MANs) 

- Digitisation 

- Document delivery services 

- Internet and the World Wide Web 

- Electronic journals 

The bids by libraries and their parent institutions for 
funding post-FoUett as well as seeking funding and 
grants from other sources have only exacerbated 
this striving for competitive advantage. This com- 
petitive culture is encouraged by the present 
Government and this is reflected in the introduction 
of league tables (Times Guide), the quadrennial 
Research Assessment Exercise, the cycle of 
Teaching Assessment visits and the FEFC inspec- 
tions. 

ACCOUNTABILITY 

Increased accountability within individual institu- 
tions is another example of political dogma at work. 
Particularly when financial resources are tight and 
getting tighter, governments and institutions are 
striving to provide value for money. In many col- 
leges and universities this is taken a stage further 
with devolved budgets and cost centres so that the 
library is accountable to academic departments for 
the provision of core and/or tailored services. 

Increasingly nowadays the library is also 
accountable directly to the students. As a body they 
have become more assertive in voicing their opin- 
ions about the services they receive. Targeted 
groups of users are also making their demands felt, 
particularly non-traditional students eg. part-time, 
overseas, disabled or distance learner etc. 

TRUE PICTURE 

So we have insecurity arising out of the political 
and educational changes taking place, competition 
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between institutions and services and accountability 
at a variety of levels within institutions. This trio 
has made it essential that mechanisms are in place 
not only to monitor and evaluate library services but 
also to maintain the high quality demanded by the 
academic community. Against this dynamic back- 
ground it can be difficult to form a true picture of 
what is being done on the ground in college and 
university libraries. Reading the professional press 
can provide a distorted view of reality. Although 
reports of research projects may be interesting in 
themselves and the results often generalisable, they 
tend to form isolated pockets of activity. Perform- 
ance assessment in recent years has been criticised 
for being hijacked by enthusiasts. They then write 
the agendas and deal with each item in turn. What’s 
wrong with that? If it weren’t for enthusiasts, not 
much would ever get off the ground! However, in 
recent years a critical mass has been reached and 
the Consultation Paper represents another step on 
that path (Joint . . , 1995, The Effective . . . ). 

Surveys 

T wo postal surveys were carried out during the 
last three years across the FE sector and the uni- 
versities. The results are provided in some detail in 
Moigan (1995) and particularly Chapter 4. The pur- 
pose of the surveys was to find out: 

- how committed institutions and their libraries 
are to performance assessment 
- which reader services are being evaluated and 
how 

- what kind of feedback is being sought from the 
academic community 

- how satisfied librarians are with their current 
procedures for evaluating their service and ask- 
ing for suggestions for improvement. 

Mth over 60% response rates in each survey the 
results could be interpreted as a fairly reliable snap- 
shot of activity. I will give details of the main 
results and then look to the common themes and 
some of the ways forward suggested by respon- 
dents. There were some surprising similarities 
between the sectors. 

RESULTS 

- Around a third of institutions have a commitment 
to performance assessment in written policies and 
documents. 

- In usage of book and periodical collections just 
over a half of the universities and over 60% of 
colleges had carried out some kind of evaluation. 
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- In the turnaround time in interlibrary loan and 
recall systems nearly 60% of universities but 
under 1 in 5 in the FE sector had carried out any 
evaluation. 

- 1 in 4 across both sectors had evaluated their 
enquiry services either quantitatively or qualita- 
tively. 

- User education was evaluated by 46% in the um- 
versity sector and 34% in the FE sector. 

- Feedback from users was sought in the usual 
ways ie. course monitoring and evaluation, 
library satisfaction surveys, discussion groups. 

- Attitude measurement is an inexact science so 
that satisfaction levels are difficult to gauge. 
Comparing respondents’ opinions and taking into 
account different expectations make for prob- 
lems. However, satisfaction levels do act as 
broad indicators. In both sectors three out of four 
libraries were dissatisfied with their procedures 
and wanted to improve the situation. 

Common Themes 

I would like to address three particular themes 
which were influential in forming responding 
librarians’ views on performance assessment: 

- Qualitative Data 
- User Expectations 
- Climate of Assessment 
QUALITATIVE DATA 

In recent years there has been a growing recognition 
in the social sciences generally that qualitative data 
have an important role to play in shedding light on 
everyday events. This move is evidenced by the 
growth in ethnographic studies, participant observa- 
tion and other types of fieldwork which yield up 
‘softer’ data. This move has been reflected in a 
number of responses from the questioimaires. 

QUANTITATIVE v QUALITATIVE 

The tenor of the argument is that statistics represent 
a limited and crude indicator and only seems to 
whet the appetite for more detail. Indeed, often sta- 
tistics pose many more questions than answers. It 
was also recognised that, to get to the heart of issues 
such as the quality of readers’ enquiry services, in- 
depth probing is required but this is labour intensive 
and time consuming. Subsequent analysis and inter- 
pretation will be equally so. Robson (1993) suggests 
that this looser approach tends to provide informa- 
tion that is enlightening for policy makers without 

4 
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the inhibiting precursor of assumptions and expecta- 
tions which may be built into hi^y structured tick 
boxes, for example. 



Figure 1 



Quantitative 


Qualitative 


Breadth/mass data 


Depth/smaller samples 


Objective 


Subjective 


‘Scientific’ 


‘Non-scientific’ 


Highly structured approach 


Looser approach 


Answers ‘how often?’ - 
statistical 


Answers ‘why’ - causative 


Less helpful with complex 


More helpful with complex 


topics 


topics 


Emphasis on neutrality 


Emphasis on the actor’s 
perspective 


Usually clear-cut precise 
results 


Useful for preliminary work 



Responses also refer to a certain snobbery about 
research methods. Crudely put, QUANTITATIVE = 
reliable, valid, rigorous real research, QUALITA- 
TIVE = unreliable, anecdotal, unscientific research. 
This is illustrated in Figure 1 with words such as 
‘objective’, ‘scientific’ and ‘neutrality’, structured 
imder the quantitative list. Very often this is based 
on personal preference or plain and simple preju- 
dice. PoUitt (1986, p.82) reminds us that ‘there is 
actually nothing inherently inferior in measures of 
feehngs or perceptions or judgements’. The more 
important question is what is done with these sub- 
jective comments to help decision-making. Much of 
performance assessment in the past has been 
focused on those areas that are easily measurable 
and quantifiable eg. economy, inputs, circulation, 
number of visits or enquiries. That was largely the 
reason for the focus. Areas such as long-term bene- 
fits of user education, quality of enquiry services, 
and availability studies are more difficult to evalu- 
ate meaningfully and are therefore less frequently 
done. The overwhelming view was that there should 
be a balance between quantitative and qualitative 
data, each complementing and supporting the other 
to provide a rich picture. 

USER EXPECTATIONS 

Another theme running through the responses was 
the problem of user expectations and how to deal 
with them. Naturally, one of the aspirations of the 



library is to satisfy the information needs of an 
increasingly diverse academic community within 
the available resources. Non-traditional students are 
a growing breed. They include students who are 
studying part-time (25%), mature students (53%), 
overseas students, those with disabilities and, 
increasingly, those studying off-campus. Thus the 
expectations of these individuals wiU form a wide 
spectrum. Library managers must perform a delicate 
balancing act to find out their needs, ensure expec- 
tations remain realistic where possible, to provide 
an appropriate range of services and assess their 
effectiveness. Many hbrary users wiU have formed 
their own views on what constitutes a ‘good’ library 
service and these views wiU act as a benchmark 
when making future judgements. Conversely, some 
users who are new to post-sixteen education may 
never have set foot in a library. Past encounters with 
hbraries and the stereotyping so beloved of the 
media may exert a heavy influence. There was an 
example in a recent TV ‘soap’ in which one char- 
acter said: ‘a librarian and interesting: isn’t that a 
contradiction in terms?’ This is proof that, although 
there are enormous changes taking place in educa- 
tion and academic libraries, some people’s attitudes 
wUl never change! 

Expectations wiU doubtless alter over the dura- 
tion of a course of study. Variables include initial 
competence and application of newly acquired skiUs 
to the hbrary’s systems, in-built self-confidence and 
the self-rehance buUt up over time, interactions with 
a variety of hbrary staff which may influence expec- 
tations for the whole service. Some students wiU 
view the library positively and optimisticaUy as an 
opportunity to make the library work for them. For 
others there may be intimidation, endless hoops to 
be jumped through - physical, organisational, Un- 
guistic or attitudinal. Some students withdraw from 
the challenge. Even fix)m students in the latter 
stages of a course there is a worryingly frequent cry 
of ‘1 haven’t got a clue where to start’. At times we 
are our own worst enemy - by raising expectations 
umealistically. We want desperately to be warm, 
friendly and helpful, the nothing-is-impossible 
approach; we are trying to be all things to aU peo- 
ple, to put over a positive image - in fact it’s not 
just customer-orientation, it’s customer-paranoia. 
The pressure to become the very antithesis of the 
stereotypical hbrarian is sometimes overwhelming. 
The advice to library staff to under-promise and 
over-deliver rather than vice versa seems sound to 
me. 1 wiU be exploring some possible solutions to 
this later. 
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CLIMATE OF REVIEW AND ASSESSMENT 

This cyclical process of not only gazing at one’s 
own navel but also having outsiders exploring the 
very same navel is obviously on the minds of 
respondents. The number and variety of review and 
assessment mechanisms is currently mind-boggling. 
As an example, take the Social Sciences Faculty 
from my own university - and this is not atypical. 
Within a three-year period the faculty will have 
taken part in six reviews - both internal and external. 
This is, of course, on top of the annual cycle of 
course monitoring and evaluation. The paperwork 
which has to be tailored to each event has become a 
nightmare. 

Respondents suggest that such periodic reviews 
can act as effective motivators in highlighting and 
encouraging reflection on how effectively the 
library supports teaching and research. However, 
when periodic equals every six months, you must 
begin to question the word motivate and substitute 
demoralise. Here are two examples where positive 
steps have been taken: 

1. One college library established a working party 
consisting of a group of local librarians who set 
out guidelines for self-inspection prior to an 
Inspectorate’s visit so as ‘to try and avoid pitfalls’. 

2. One college librarian (Davies, 1994) took an 
interesting further step by becoming a lay inspec- 
tor with the FEFC. He detailed the training under- 
taken and views optimistically the prospects of 
college libraries within the new framework. 

Imminent inspections, visits from professional 
bodies, course validations, franchise agreements, 
visits from Funding Councils and the Higher 
Education Quality Council all provide opportunities 
to ensure that appropriate performance assessment 
activities are in place and functioning effectively. 
It’s funny how time is always found when push 
comes to shove! Such events have succeeded in 
either initiating and providing the impetus or fine- 
tuning existing review and assessment procedures. 

Recent reports (Joint Funding Councils, 1995; 
Joint Funding Councils’ Libraries Review Group, 
1993; 1995) have all recognised the need for a 
tighter fit between library services and internal/ 
external assessment. 

The quote taken from a university librarian has 
become an understatement for some institutions: 
‘We must be careful not to spend so much time 
measuring how we do the job that we detract from 
our performance’. 







Suggestions for Possible Ways 
Forward 

J have divided up suggestions into six categories: 

- Greater involvement of a variety of constituents 

- More systematic and structured approach 

- Greater proactivity 

- Wider application of IT 

- Greater emphasis on the user perspective 

- More time 

GREATER INVOLVEMENT OF A VARIETY OF 
CONSTITUENTS 

Library Staff 

The participation of large numbers of library staff in 
any performance assessment activity will depend on 
the size of the library, the number of staff, the views 
of the library’s management, the staffing structure 
and organisation, the culture etc. Performance 
assessment, in whatever form it takes, needs to form 
a recognised part of the service management not as 
an isolated one-off event. It needs to be internalised 
within the library and the wider institution. If the 
library is run in a participative way, this process will 
be made easier. Some library managers may be 
reluctant to become heavily involved in these activi- 
ties due to a lack of confidence in or unfamiliarity 
with appropriate research or evaluation methods. 
Such methods may be alien to some. However, one 
option would be to call on the expertise of academic 
colleagues, as outlined below. Alternatively, a set of 
basic research methods can be gleaned from suitable 
textbooks. 

Library Users 

This group is important enough to have its own sec- 
tion below. 

Academic Staff 

Respondents suggested that often libraries can be 
somewhat isolationist in their approach to evaluat- 
ing their services. Why are we afraid to call upon 
the expertise of our own academic colleagues? 
Academics generally are only too willing to share 
their knowledge and skills and apply them within a 
library setting. This is particularly the case when it 
is so obviously aimed at improving services. Areas 
for possible collaboration are many: computing and 
technology, social science research methods, market 
research, psychological testing, assessment in teach- 
ing and learning etc. 
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Other Libraries 

Respondents gave examples of informal gatherings 
of librarians to hammer out guidelines as well as 
providing mutual support. Also mentioned was an 
informal system of library peer review between 
institutions in a particular region or locality. This is 
an approach that is based on co-operation, support, 
learning by mistakes, strength in numbers and on 
the notion that outsiders can often see things that 
insiders caimot. 

MORE SYSTEMATIC AND STRUCTURED 
APPROACH 

In a sense this has been partly taken over by events 
with the Consultation Document (Joint, 1995, The 
Effective . . .) and also the work being carried out by 
the Council for Learning Resources in Colleges 
(CoLRiC). Much of the groundwork has been car- 
ried out particularly through SCONUL and its 
Advisory Committee on Performance Indicators. 

The suggested framework provides a useful practi- 
cal foundation on which to build, perhaps even 
more widely than just the university sector. This 
‘rich picture’ approach is the way forward although 
certain aspects concerning satisfaction levels require 
modiftcations. This is a very welcome document. 

GREATER PROACTIVITY 

Advice which was given by a number of respon- 
dents concerned the issue of librarians drawing up 
their own agenda. The suggestion was that if the 
library managers do not take the initiative, someone 
else wiU. The someone else will be part of the par- 
ent organisation’s management and will have his/her 
own views on how the library service shoirld be 
evaluated and developed. Take the lead! Get in the 
driver’s seat! It’s too late to whinge once events 
have moved on and management is demanding effi- 
ciency gains in areas which, if you had been the 
decision-maker, you woirld not be considering. The 
various quality audit, assessment and inspection 
events provide ideal vehicles by which libraries can 
- if they do not already - start to integrate into the 
parent organisation and establish performance 
assessment mechanisms. Such activity has the dual 
by-products of increasing the library’s credibility 
and improving the likelihood of success in future 
requests for additional funding. 

WIDER APPLICATION OF IT 

There are a variety of software packages, increas- 
ingly sophisticated developments within integrated 
library computer systems and other management 
information systems to help with evaluation of ser- 



vices. Respondents suggested that automated sys- 
tems nowadays have the capability of providing 
ever more sophisticated and disaggregated informa- 
tion about library usage etc. Lack of priority, lack of 
time, lack of staff and funds - or any combination - 
will be influencing factors. It is obvious from the 
titles of some of the seminars and poster sessions at 
this Conference that there are exciting projects - 
some more longstanding than others - involving IT 
and particirlarly qualitative data. With that I will 
leave IT and performance assessment to those who 
know! 

GREATER EMPHASIS ON USER PERSPECTIVES 

The postal surveys - not unsirrprisingly - have high- 
lighted the importance of the user, customer, con- 
sumer, client, patron or the awfully named end-user. 
Customer-orientation has become a veritable indus- 
try in its own right. Involving the users from the 
beginning in planning services right through to 
monitoring and evaluating them is now accepted 
practice. In many UK libraries 15 years ago this 
involvement woirld have appeared revolutionary, 
rather bizarre and urmecessary. The prevailing ethos 
is summed up by Booth (1993, p.7): 

‘If you are providing a service, there will always 

be a subjective element in assessing quality; and 

... the most important judge is the customer’. 

For example, it would be difficult to conceive of the 
evaluation of a readers’ enquiry service without 
some recognition of the quality, accuracy and com- 
petence of the service from the enquirer’s point of 
view. 

So how should we place more emphasis on the 
users’ perspectives? 

1. Increasing Adoption of User Survey 

User surveys can be particularly helpful in evaluat- 
ing services either focusing on specific elements of 
service provision eg. services to part-time or other 
non-traditional students, undergraduate or short loan 
collections, use of periodicals, satisfaction with user 
education or enquiry services. In this way it is pos- 
sible to avoid the dangers of survey or questiormaire 
fatigue and alienate the very people whose views 
you are seeking. Alternatively, the library could 
carry out a rolling programme of surveys which 
covers major services but takes place every two or 
three years. 

Packages such as LIBRA from Priority Search - 
which some of you will be familiar with - can help 
to gain qualitative feedback from the academic 
community through IT-related means. 
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2. Charterism 

Although the present UK Government’s Citizen’s 
Charter programme - there are currently over 40 
published Government charters - has been discredit- 
ed in some quarters, the principles underlying the 
movement are being played out in academic institu- 
tions and many of their libraries. These principles 
include words like ‘standards’ and ‘value for 
money’ which most people wUl be in favour of. 
Nowadays many libraries support charters or rights 
and responsibilities which display realistic expecta- 
tions: what users can expect from the library and 
what the library can expect from its users. Built into 
some of these charters are commitments to certain 
standards eg. obtaining available material from 
other c am pus libraries within three working days. 

Charterism may also have the knock-on effects of 
improving communication with the academic com- 
munity, demonstrating a commitment to quality and 
focusing the attention of library staff on specific 
service issues. 

3. Customer Care Programmes 

Although out of a similar stable to Charterism, cus- 
tomer care programmes can be viewed rather cyni- 
cally as being superficial and cosmetic. This is natu- 
rally a debatable point and depends largely on the 
organisational culture and the attitudes of individual 
members of staff. What these training programmes 
and customer care statements have succeeded in 
doing is highlighting the importance of the user. In 
general terms I would put its significance as no 
greater than that. 

4 ‘Making the familiar strange’ 

This is a perspective taken from the sociological 
toolkit (Delamont, 1981). It is a genuine attempt to 
try to understand the users’ needs and views by 
looking at the service from their perspective. It may 
be necessary at times to challenge the taken-for- 
granted ideas and attitudes that librarians have built 
up over the years and see through the eyes of the 
user. It was originally used in researching school 
classroom activity where Delamont highlighted the 
difficulty of seeing in a different light the normal, 
the usual, the everyday, the ordinary, the common- 
place, the routine. To gain possibly novel insights 
into the teaching and learning activities - or library 
services in our case - it may be necessary to ‘make 
the familiar strange’. Two examples would be: 

- To study, observe or participate in non-library 
settings which are chosen for their parallel fea- 
tures eg. attendance at lectures, seminars; 
observing other service providers eg. doctors. 







dentists, shops, civil service operations. 

Citizens’ Advice Bureaux, tourist offices etc. 
This forces the librarian to regard services and 
their quality from the viewpoint of the recipient. 

- To make use of other libraries and carry out 
some of the procedures that students would be 
expected to carry out eg. accessibility, availabil- 
ity, use of library catalogues, effectiveness of 
guiding and signposting etc. 

A^^thout wishing to overstate the case, these kinds of 
activities are intended to illustrate the potential for 
avoiding the perpetuation of misguided views. 

MORE TIME 

It has to be recognised that performance assessment 
is an integral part of library management and needs 
to be accorded time or at least to rise a few notches 
up the priority list - as when external events force 
the issue. Respondents felt imder such tremendous 
pressure to provide a high quality service that they 
had precious little time to judge the effectiveness of 
that service. It is ironic that the financial squeeze 
and the increase in student numbers mean that fund- 
ing has to be carefuUy targeted, quality has to be 
maintained and yet librarians have less and less time 
to find out how well the strategies are working. 

Conclusions 

I t is encouraging to hear the optimism of those sur- 
veyed. A large number recognise the importance 
of evaluating services but in practice feel frustrated 
for all sorts of reasons. However, they were willing 
to make a number of practical suggestions for ways 
forward. In a number of cases nationally, pockets of 
activity have turned into larger co-operative projects 
and co-ordinated approaches which augur well for 
the next five years. 
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